NumPy and J make Sweet Array Love

Import NumPy using the standard naming convention


In [1]:
import numpy as np

Configure the J Python3 addon

To use the J Python3 addon you must edit path variables in jbase.py so Python can locate the J binaries. On my system I set:

# typical for windows install in home
pathbin= 'c:/j64/j64-807/bin'
pathdll= pathbin+'/j.dll'
pathpro= pathbin+'/profile.ijs'

Insure jbase.py and jcore.py are on Python's search path


In [2]:
import sys

# local api/python3 path - adjust path for your system
japipath = 'C:\\j64\\j64-807\\addons\\api\\python3'

if japipath not in sys.path:
    sys.path.append(japipath)

sys.path


Out[2]:
['',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\python36.zip',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\DLLs',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\site-packages',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\site-packages\\win32',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\site-packages\\win32\\lib',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\site-packages\\Pythonwin',
 'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\site-packages\\IPython\\extensions',
 'C:\\Users\\jbaker\\.ipython',
 'C:\\j64\\j64-807\\addons\\api\\python3']

In [3]:
import jbase as j 
print(j.__doc__)


usage:
 jbase.init(True)  # True (default) loads profile - False avoids
 
 jbase.dor('i.2 3 4')    # run sentence and print output result
 jbase.do(('+a.')        # run and return error code
 jbase.getr()            # get last output result
 jbase.do('abc=: i.2 3') # define abc
 q= jbase.get('abc')     # get q as numpy array from J array 
 jbase.set('ghi',23+q)   # set J array from numpy array
 jbase.jdor('ghi')       # print array 
 jbase.j()               # J repl - .... to exit

types:
 python types supported: strings, bytes, numpy int64/float64 
 numpy arrays have shape
 json covers some other requirements
 
Developed with Python 3.6.4 (Anaconda) and J807.

Works with python kernel in Jupyiter.



In [4]:
# start J - only one instance currently allowed
try:
    j.init()
except:
    print('j running')

In [5]:
j.dor("i. 2 3 4")  # run sentence and print output result


 0  1  2  3
 4  5  6  7
 8  9 10 11

12 13 14 15
16 17 18 19
20 21 22 23

In [6]:
rc = j.do(('+a.'))  # run and return error code
print(rc)
j.getr()            # get last output result


3
Out[6]:
'|domain error\n|       +a.\n'

In [7]:
j.do('abc=: i.2 3') # define abc
q= j.get('abc')     # get q as numpy array from J array 
print (q)


[[0 1 2]
 [3 4 5]]

In [8]:
j.set('ghi',23+q)   # set J array from numpy array
j.dor('ghi')        # print array (note typo in addon (j.__doc___)


23 24 25
26 27 28

Character data is passed as bytes.


In [9]:
j.do("cows =. 'don''t have a cow man'")
j.get('cows')


Out[9]:
b"don't have a cow man"

In [10]:
ido = "I do what I do because I am what I am!"
j.set("ido", ido)
j.dor("ido")


I do what I do because I am what I am!

j.j() enters a simple REPL

Running j.j() opens a simple read, execute and reply loop with J. Exit by typing ....


In [11]:
# decomment to run REPL 

# j.j()

J accepts a subset of NumPy datatypes

Passing datatypes that do not match the types the J addon supports is allowed but does not work as you might expect.


In [12]:
# boolean numpy array
p = np.array([True, False, True, True]).reshape(2,2)

In [13]:
p


Out[13]:
array([[ True, False],
       [ True,  True]])

In [14]:
j.set("p", p)

In [15]:
j.dor("p")


8.32143e_317 1.11493e_311
           0 2.97079e_313

As you can see a round trip of numpy booleans generates digital noise.

The only numpy datatypes J natively supports on Win64 systems are:

  1. np.int64

  2. np.float64

  3. simple character strings - passed as bytes

To use other types it will be necessary to encode and decode them with Python and J helper functions. The limited datatype support is not as limiting as you might expect. The default NumPy array is np.float64 on 64 bit systems and the majority of NumPy based packages manipulate floating point and integer arrays.

NumPy and J are derivative Iverson Array Processing Notations

The following NumPy examples are from the SciPy.org's NumPy quick start tutorial. For each NumPy statement, I have provided a J equivalent

Creating simple arrays


In [16]:
# numpy
a = np.arange(15).reshape(3, 5)
print(a)


[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]]

In [17]:
# J
j.do("a =. 3 5 $ i. 15")
j.dor("a")


 0  1  2  3  4
 5  6  7  8  9
10 11 12 13 14

In [18]:
# numpy
a = np.array([2,3,4])
print(a)


[2 3 4]

In [19]:
# J
j.do("a =. 2 3 4")
j.dor("a")


2 3 4

In [20]:
# numpy
b = np.array([(1.5,2,3), (4,5,6)])
print(b)


[[1.5 2.  3. ]
 [4.  5.  6. ]]

In [21]:
# J
j.do("b =. 1.5 2 3 ,: 4 5 6")
j.dor("b")


1.5 2 3
  4 5 6

In [22]:
# numpy
c = np.array( [ [1,2], [3,4] ], dtype=complex )
print(c)


[[1.+0.j 2.+0.j]
 [3.+0.j 4.+0.j]]

In [23]:
# J
j.do("c =. 0 j.~ 1 2 ,: 3 4")
j.dor("c")           # does not show as complex
j.dor("datatype c")  # c is complex


1 2
3 4
complex

In [24]:
# numpy - make complex numbers with nonzero real and imaginary parts
c + (0+4.7j)


Out[24]:
array([[1.+4.7j, 2.+4.7j],
       [3.+4.7j, 4.+4.7j]])

In [25]:
# J - also for J
j.dor("c + 0j4.7")


1j4.7 2j4.7
3j4.7 4j4.7

In [26]:
# numpy
np.zeros( (3,4) )


Out[26]:
array([[0., 0., 0., 0.],
       [0., 0., 0., 0.],
       [0., 0., 0., 0.]])

In [27]:
# J
j.dor("3 4 $ 0")


0 0 0 0
0 0 0 0
0 0 0 0

In [28]:
# numpy - allocates array with whatever is in memory
np.empty( (2,3) )


Out[28]:
array([[1.5, 2. , 3. ],
       [4. , 5. , 6. ]])

In [29]:
# J - uses fill - safer but slower than numpy's trust memory method
j.dor("2 3 $ 0.0001")


0.0001 0.0001 0.0001
0.0001 0.0001 0.0001

Basic Operations


In [30]:
# numpy
a = np.array( [20,30,40,50] )
b = np.arange( 4 )
c = a - b
print(c)


[20 29 38 47]

In [31]:
# J
j.do("a =. 20 30 40 50")
j.do("b =. i. 4")
j.do("c =. a - b")
j.dor("c")


20 29 38 47

In [32]:
# numpy - uses previously defined (b)
b ** 2


Out[32]:
array([0, 1, 4, 9], dtype=int32)

In [33]:
# J
j.dor("b ^ 2")


0 1 4 9

In [34]:
# numpy - uses previously defined (a)
10 * np.sin(a)


Out[34]:
array([ 9.12945251, -9.88031624,  7.4511316 , -2.62374854])

In [35]:
# J
j.dor("10 * 1 o. a")


9.12945 _9.88032 7.45113 _2.62375

In [36]:
# numpy - booleans are True and False
a < 35


Out[36]:
array([ True,  True, False, False])

In [37]:
# J - booleans are 1 and 0
j.dor("a < 35")


1 1 0 0

Array Processing


In [38]:
# numpy
a = np.array( [[1,1], [0,1]] )
b = np.array( [[2,0], [3,4]] )

# elementwise product
a * b


Out[38]:
array([[2, 0],
       [0, 4]])

In [39]:
# J
j.do("a =. 1 1 ,: 0 1")
j.do("b =. 2 0 ,: 3 4")
j.dor("a * b")


2 0
0 4

In [40]:
# numpy - matrix product
np.dot(a, b)


Out[40]:
array([[5, 4],
       [3, 4]])

In [41]:
# J - matrix product
j.dor("a +/ . * b")


5 4
3 4

In [42]:
# numpy - uniform pseudo random - seeds are different in Python and J processes - results will differ
a = np.random.random( (2,3) )
print(a)


[[0.08372239 0.37105675 0.2239006 ]
 [0.07790648 0.32783563 0.09191487]]

In [43]:
# J - uniform pseudo random
j.dor("?. 2 3 $ 0")


0.038363  0.329284 0.335644
0.985972 0.0583756 0.282326

In [44]:
# numpy - sum all array elements - implicit ravel
a = np.arange(100).reshape(20,5)
a.sum()


Out[44]:
4950

In [45]:
# j - sum all array elements - explicit ravel
j.dor("+/ , 20 5 $ i.100")


4950

In [46]:
# numpy
b = np.arange(12).reshape(3,4)
print(b)
# sum of each column
print(b.sum(axis=0))
# min of each row
print(b.min(axis=1))
# cumulative sum along each row
print(b.cumsum(axis=1))
# transpose
print(b.T)


[[ 0  1  2  3]
 [ 4  5  6  7]
 [ 8  9 10 11]]
[12 15 18 21]
[0 4 8]
[[ 0  1  3  6]
 [ 4  9 15 22]
 [ 8 17 27 38]]
[[ 0  4  8]
 [ 1  5  9]
 [ 2  6 10]
 [ 3  7 11]]

In [47]:
# J 
j.do("b =. 3 4 $ i. 12")
j.dor("b")
# sum of each column
j.dor("+/ b")
# min of each row
j.dor('<./"1 b')
# cumulative sum along each row
j.dor('+/\\"0 1 b')   # must escape \ character to pass +/\"0 1 properly to J
# transpose
j.dor("|: b")


0 1  2  3
4 5  6  7
8 9 10 11
12 15 18 21
0 4 8
0  1  3  6
4  9 15 22
8 17 27 38
0 4  8
1 5  9
2 6 10
3 7 11

Indexing and Slicing


In [48]:
# numpy 
a = np.arange(10) ** 3 
print(a[2])
print(a[2:5])
print(a[ : :-1])   # reversal


8
[ 8 27 64]
[729 512 343 216 125  64  27   8   1   0]

In [49]:
# J
j.do("a =. (i. 10) ^ 3")
j.dor("2 { a")
j.dor("(2 + i. 3) { a")
j.dor("|. a")


8
8 27 64
729 512 343 216 125 64 27 8 1 0

Passing Larger Arrays

Toy interfaces abound. Useful interfaces scale. The current addon is capable of passing large enough arrays for serious work. Useful subsets of J and NumPy arrays can be memory mapped. It wouldn't be difficult to memory map very large (gigabyte sized) NumPy arrays for J.


In [50]:
from numpy import pi
x = np.linspace( 0, 2*pi, 100, np.float64) # useful to evaluate function at lots of points
f = np.sin(x)
f


Out[50]:
array([ 0.00000000e+00,  6.34239197e-02,  1.26592454e-01,  1.89251244e-01,
        2.51147987e-01,  3.12033446e-01,  3.71662456e-01,  4.29794912e-01,
        4.86196736e-01,  5.40640817e-01,  5.92907929e-01,  6.42787610e-01,
        6.90079011e-01,  7.34591709e-01,  7.76146464e-01,  8.14575952e-01,
        8.49725430e-01,  8.81453363e-01,  9.09631995e-01,  9.34147860e-01,
        9.54902241e-01,  9.71811568e-01,  9.84807753e-01,  9.93838464e-01,
        9.98867339e-01,  9.99874128e-01,  9.96854776e-01,  9.89821442e-01,
        9.78802446e-01,  9.63842159e-01,  9.45000819e-01,  9.22354294e-01,
        8.95993774e-01,  8.66025404e-01,  8.32569855e-01,  7.95761841e-01,
        7.55749574e-01,  7.12694171e-01,  6.66769001e-01,  6.18158986e-01,
        5.67059864e-01,  5.13677392e-01,  4.58226522e-01,  4.00930535e-01,
        3.42020143e-01,  2.81732557e-01,  2.20310533e-01,  1.58001396e-01,
        9.50560433e-02,  3.17279335e-02, -3.17279335e-02, -9.50560433e-02,
       -1.58001396e-01, -2.20310533e-01, -2.81732557e-01, -3.42020143e-01,
       -4.00930535e-01, -4.58226522e-01, -5.13677392e-01, -5.67059864e-01,
       -6.18158986e-01, -6.66769001e-01, -7.12694171e-01, -7.55749574e-01,
       -7.95761841e-01, -8.32569855e-01, -8.66025404e-01, -8.95993774e-01,
       -9.22354294e-01, -9.45000819e-01, -9.63842159e-01, -9.78802446e-01,
       -9.89821442e-01, -9.96854776e-01, -9.99874128e-01, -9.98867339e-01,
       -9.93838464e-01, -9.84807753e-01, -9.71811568e-01, -9.54902241e-01,
       -9.34147860e-01, -9.09631995e-01, -8.81453363e-01, -8.49725430e-01,
       -8.14575952e-01, -7.76146464e-01, -7.34591709e-01, -6.90079011e-01,
       -6.42787610e-01, -5.92907929e-01, -5.40640817e-01, -4.86196736e-01,
       -4.29794912e-01, -3.71662456e-01, -3.12033446e-01, -2.51147987e-01,
       -1.89251244e-01, -1.26592454e-01, -6.34239197e-02, -2.44929360e-16])

In [51]:
j.set("f", f)

In [52]:
j.get("f")


Out[52]:
array([ 0.00000000e+00,  6.34239197e-02,  1.26592454e-01,  1.89251244e-01,
        2.51147987e-01,  3.12033446e-01,  3.71662456e-01,  4.29794912e-01,
        4.86196736e-01,  5.40640817e-01,  5.92907929e-01,  6.42787610e-01,
        6.90079011e-01,  7.34591709e-01,  7.76146464e-01,  8.14575952e-01,
        8.49725430e-01,  8.81453363e-01,  9.09631995e-01,  9.34147860e-01,
        9.54902241e-01,  9.71811568e-01,  9.84807753e-01,  9.93838464e-01,
        9.98867339e-01,  9.99874128e-01,  9.96854776e-01,  9.89821442e-01,
        9.78802446e-01,  9.63842159e-01,  9.45000819e-01,  9.22354294e-01,
        8.95993774e-01,  8.66025404e-01,  8.32569855e-01,  7.95761841e-01,
        7.55749574e-01,  7.12694171e-01,  6.66769001e-01,  6.18158986e-01,
        5.67059864e-01,  5.13677392e-01,  4.58226522e-01,  4.00930535e-01,
        3.42020143e-01,  2.81732557e-01,  2.20310533e-01,  1.58001396e-01,
        9.50560433e-02,  3.17279335e-02, -3.17279335e-02, -9.50560433e-02,
       -1.58001396e-01, -2.20310533e-01, -2.81732557e-01, -3.42020143e-01,
       -4.00930535e-01, -4.58226522e-01, -5.13677392e-01, -5.67059864e-01,
       -6.18158986e-01, -6.66769001e-01, -7.12694171e-01, -7.55749574e-01,
       -7.95761841e-01, -8.32569855e-01, -8.66025404e-01, -8.95993774e-01,
       -9.22354294e-01, -9.45000819e-01, -9.63842159e-01, -9.78802446e-01,
       -9.89821442e-01, -9.96854776e-01, -9.99874128e-01, -9.98867339e-01,
       -9.93838464e-01, -9.84807753e-01, -9.71811568e-01, -9.54902241e-01,
       -9.34147860e-01, -9.09631995e-01, -8.81453363e-01, -8.49725430e-01,
       -8.14575952e-01, -7.76146464e-01, -7.34591709e-01, -6.90079011e-01,
       -6.42787610e-01, -5.92907929e-01, -5.40640817e-01, -4.86196736e-01,
       -4.29794912e-01, -3.71662456e-01, -3.12033446e-01, -2.51147987e-01,
       -1.89251244e-01, -1.26592454e-01, -6.34239197e-02, -2.44929360e-16])

In [53]:
r = np.random.random((2000,3000))
r = np.asarray(r, dtype=np.float64)

In [54]:
r


Out[54]:
array([[0.53707077, 0.60373505, 0.52419947, ..., 0.70380001, 0.9123579 ,
        0.05837318],
       [0.59133912, 0.86656977, 0.34583369, ..., 0.03890694, 0.16030787,
        0.20382021],
       [0.71522825, 0.46182252, 0.91629691, ..., 0.28795434, 0.57352725,
        0.35592709],
       ...,
       [0.06881621, 0.03503802, 0.90719109, ..., 0.69769166, 0.98276971,
        0.35669076],
       [0.15146427, 0.89686607, 0.61242936, ..., 0.83553244, 0.45396481,
        0.65823819],
       [0.50264795, 0.40157774, 0.13658319, ..., 0.07277537, 0.70719058,
        0.6776694 ]])

In [55]:
j.set("r", r)

In [56]:
j.get("r")


Out[56]:
array([[0.53707077, 0.60373505, 0.52419947, ..., 0.70380001, 0.9123579 ,
        0.05837318],
       [0.59133912, 0.86656977, 0.34583369, ..., 0.03890694, 0.16030787,
        0.20382021],
       [0.71522825, 0.46182252, 0.91629691, ..., 0.28795434, 0.57352725,
        0.35592709],
       ...,
       [0.06881621, 0.03503802, 0.90719109, ..., 0.69769166, 0.98276971,
        0.35669076],
       [0.15146427, 0.89686607, 0.61242936, ..., 0.83553244, 0.45396481,
        0.65823819],
       [0.50264795, 0.40157774, 0.13658319, ..., 0.07277537, 0.70719058,
        0.6776694 ]])

In [57]:
r.shape


Out[57]:
(2000, 3000)

In [58]:
j.get("r").shape


Out[58]:
(2000, 3000)

In [59]:
j.dor("r=. ,r")
j.get("r").shape


Out[59]:
(6000000,)

In [60]:
r.sum()


Out[60]:
2999717.9025033144

In [61]:
b = np.ones((5,300,4), dtype=np.int64)
j.set("b", b)
b2 = j.get("b")
print(b.sum())
print(b2.sum())


6000
6000

Some NumPy goodies of interest to J programmers

Work in Progress - stay tuned.